Okay.

Welcome to the last week of AI2.

I want to conclude this chapter on natural language processing with current methods,

i.e. deep learning methods.

We had talked about why language might be interesting and what might be difficult

about language.

We had talked about classical, i.e. mostly statistical and symbolic methods for natural

language processing and now comes the current hype deep learning methods for

natural language processing.

So, the problem, of course, if we want to use symbolic methods, then we essentially

craft or hand craft rules.

That has two limitations.

One is we humans have to understand what language does in order to craft the rules.

The other thing is we have to write them down so they become computationally inefficient

and don't interact adversely with each other and so on.

Of course, wouldn't it be nice if we could just learn this stuff?

That's really what we want to look at now.

We have lots of text.

The internet rose by 100 billion words a day.

It's been doing that for quite a while.

We have loads of text.

We are going to look at what we can do with data and deep learning methods in this space.

When I have been preparing this, I have been a bit disappointed because it's still.

Lots of nice pictures, but there is a lot of voodoo in it.

How do you do these things and why that actually may work?

It's undeniable that these methods do work.

They work quite well and in some cases in my mind, unreasonably well.

That's very nice.

The first kind of topic I want to look into is word embeddings.

The idea is that we have neural networks.

They actually want numbers.

Excitation.

Remember that a neuron is something that takes a vector of inputs.

Then waits them, computes the weighted sum, puts some kind of a sigmoid or other activation

function on top of it and sends out a signal or not, depending on whether the weighted

sum actually meets the threshold or not.

We have these neural networks that essentially need numbers as inputs.

One of the problems when doing languages, how do we make language into vectors, sequences

of small real numbers?

We've already seen one of such techniques in information, which we evolved, which is

just the classic one-hot vector.

If you have a word, then you add in the right dimension of one.

The same word.

Another time you add another one.

Stuff like that.

It turns out that even though we kind of went to TFI, the F-pectors, we can do better.

And how to do better is kind of the first topic.

How to make words into vectors.

There was a classical first approach word to veck, which is something you can still download

and it's still useful, but we have better words and vetting by now.